04. Internal Components of an RL Trading Agent - Policy

Understanding Agent Actions in Q-Learning

Designing intelligent agents involves implementing a policy dictating when actions are chosen, termed the act function:

• State Input, Action Output: The act function takes a state and outputs an action.

• ε-Greedy Strategy:

  • Introduces randomness; with a probability of ε, agents pick random actions.
  • Encourages exploration while still exploiting known information with a DQN (Deep Q-Network).

• Off-Policy Learning:

  • Differentiates behavior policy (for action selection) and target policy (for learning improvement), using the Bellman equation.

• Action Selection Process:

  1. Generate a random number (0-100).
  2. Compare this with ε value.
  3. If below ε, select a random action; otherwise, use DQN for the best action.

• Experience Replay:

  • Enhances learning by revisiting past state-action pairs.
  • Utilizes a mini-batch to refine target Q-values and neural network fitting for policy enhancement.

Consistent iteration of this mechanism improves the agent's performance over time.

Supplementary Material - On- vs. Off-Policy Learning

The Following Medium article gives a good description of on- and off-policy learning. It also gees through a different strategy for Off-Policy learning called "Important Sampling". This will not be used in this course, as we use Q-learning as our off-policy learning technique.

On-Policy v/s Off-Policy Learning

Supplementary Material - Exploration, Exploitation, and ε-greedy Learning

The following Medium article gives a deep-dive into the exploration-exploitation dilemma and how the ε-greedy strategy is used to address this dilemma.

RL Series#3: To explore or not to explore, that is the question